DOCUMENT RESUME 



ED 441 858 



TM 030 893 



AUTHOR 

TITLE 

SPONS AGENCY 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Donald, Janet G. 

Indicators of Success: From Concepts to Classrooms. 

Canadian Social Science and Humanities Research Council, 
Ottawa (Ontario) . 

2000-04-25 

28p.; Paper presented at the Annual Meeting of the American 
Educational Research Association (New Orleans, LA, April 
24-28, 2000) . Also funded by the Quebec Fonds pour la 
Formation de Chercheurs et I'Aide a la Recherche. 

Reports - Research (143) - - Speeches /Meeting Papers (150) 

MF01/PC02 Plus Postage. 

*College Faculty; Educational Research; Foreign Countries; 
Higher Education; *Learning; *Prof essional Development; 
Research Methodology; *Success; *Teacher Education 
* Educational Indicators 



ABSTRACT 



This paper has three objectives: (1) to examine the context 

in which faculty developers operate; (2) to review 2 5 -year program of 
research for the findings that are most pertinent to teacher developers; and 
(3) to situate research into teacher development and consider the strategies 
that might persuade others to conduct research into teacher development in 
postsecondary education. The paper begins with an investigation of the 
structure of knowledge in university courses and moved through studies of the 
expectations of learning held by professors and students in selected 
disciplines. The results of the program of investigations over the years fall 
into three nested categories. At the most general level are findings about 
methods and lines of convergence across institutions and disciplines. At a 
more specific level is the examination of student learning and the 
comparisons between different types of student experience. Nested within 
these areas is the study of effects of differences in learning contexts. From 
this body of research, certain indicators of success for faculty developers 
have been derived. These are: (1) appropriate content in a course; (2) 

whether or not students have learned to think in a new way; and (3) the 
extent to which students incorporate their learning from courses into their 
other studies and their lives as scholars. A second set of indicators 
revolves around clear expectations for students and strategies for learning. 

A third set of indicators is needed to guide the research of faculty 
developers, and to explain instructional goals and methods and the extent to 
which they can be transferred. (Contains 2 tables, 2 figures and 29 
references.) (SLD) . 
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As faculty developers we find ourselves called upon to fulfill a variety of roles - 
orienting new faculty, guide, evaluator, researcher, empathizer. These roles are carried 
out in institutions that often provide minimal structure, so that we are also explorers - 
testers of deep water, with Teaching Tips (McKeachie, 1999) as our scripture. How can 
we know when we have been successful? 

I have three objectives in this address. The first is to examine the context in 
which we as faculty developers operate. This is important because if we are to be 
effective as faculty developers, we need to understand the conditions that frame our 
work, and the limits and possibilities that these conditions oblige. Identifying the context 
is the first defining characteristic of expertise, a proviso for knowing ourselves (Donald, 
1992). The second objective is to examine my twenty-five year program of research for 
those findings that are most pertinent to our metier as faculty developers. The third 
objective originates in our reform legacy as psychologists and educators; it is to situate 
where we might be going and what strategies we might use to persuade others to 
accompany us. Each objective has a particular relation to indicators of success in 
postsecondary education. 

The context in which we operate 

Universities and colleges have a range of purposes and organizational 
characteristics, but have certain attributes in common that determine what we can 
accomplish as faculty developers. The attributes include both goals and limitations. One 
goal is axiomatic to our discussion: the ultimate outcome of instructional practice is 
effective student learning. Over the years I have become increasingly aware that the 
faculty evaluation process we have developed in our universities focuses attention on 
an important part of the student learning experience, but does not take into account 
other factors that have major effects on student learning. These factors , frequently 
ignored, have major and at times insidious effects on what and how students learn. One 
factor is the diffuseness and complexity of the educational environment in a 



Indicators of success - AERA 2000 3 



postsecondary institution. Another is the variety of knowledge bases, skills, attitudes, 
and epistemologies our students bring to the learning milieu. A third is the longstanding 
assumption that students learn by attending lectures in large groups; this pervasive 
assumption has led decision makers in the institution to ignore the learning process. 
These factors are limiting conditions for faculty development and pedagogical 
improvement. Their effects are shown in the following practices. 

First, it has been common practice to evaluate the learning experience of our 
students classroom by classroom. The learning environment of our students, however, 
consists of the entire campus, and with the advent of the internet, increasingly extends 
beyond campus. This means that learning experiences are increasingly variable. On 
campus, students will spend less than three hours per week in any given classroom, 
and other venues such as the library, laboratory, cafeteria, work or field placements, or 
the student’s own room acquire greater importance as learning settings. Undergraduate 
students report that their learning takes place when they do their assignments, not in 
the classroom (Donald & Denison, 1997). The point of contact for pedagogical 
improvement is therefore difficult to find. Considering the limited time students spend in 
classrooms, evaluating what has happened in a classroom may capture at most the 
lion’s whisker. 

In our evaluation of instruction, we ask our students to provide indices of their 
satisfaction with their learning situation. Increasingly, we factor into the analysis some 
measure of their background and their achievement. But we are not proactive in 
establishing their needs and the appropriate learning situations to respond to them; 
needs assessments at the beginning of courses are rare. We have also neglected to 
measure students’ insight into their responsibility for learning and their preparation for 
independent learning. We do not tend, for example, to have a contract for learning that 
spells out the obligations of all parties. In my institution, the students have a contract 
with the university, and professors have a contract with the university, but professors 
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and students do not have a contract with each other, nor do programs have contracts. 
The terms of instruction and learning outcomes not stated. 

The effect of these missing links is exacerbated by the fact that students are 
frequently newly independent and testing their independence from family and home 
town constraints. Added to the equation, students often lack the strategies needed for 
independent learning, so their own desired independence may work against success in 
learning. I have argued elsewhere (Donald, 2000) that this problem requires a set of 
strategies on the part of instructors to enlighten students on their role as independent 
learners. The largest contributor to student learning gains at the postsecondary level is 
the quality of effort students put into their work (Pace, 1988). To date, however, 
institutions have not been highly successful in getting across this message. On the 
contrary, students who have been enculturated to be consumers, extend this role in 
relation to the university, which relieves them of the responsibility for learning. 

Given these limitations, the institution places instructors and us as faculty 
developers in a dilemma of major proportions - instructors are evaluated for learning 
situations over which they have relatively little control, by undergraduates who may lack 
a clear understanding of what their responsibilities as learners are. In response to this 
situation, I have suggested that individual professors can take certain steps, for 
example, by explaining educational goals, supplying an overview of their discipline, and 
clarifying to students that their learning will depend primarily upon the quality of effort 
they put into their work (Donald, 2000). But I think our role as faculty developers places 
the onus on us to determine in what ways we can improve the match between student 
learning and accountability for that learning. What has my research as a faculty 
developer told me? 

Research on learning in the university 

A major impetus for my research has come from the sociopolitical level of 
discourse, where the focal issues have been the quality and evaluation of teaching, 
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(Donald, 1991a; 1996; Donald & Denison, 1996; Donald, Saroyan & Denison, 1995). 
The foundation for my research agenda, however, is higher order learning, how we help 
postsecondary students to think. In order to answer the questions that intrigued me 
about postsecondary teaching and learning, I have analyzed professors' knowledge 
structures and students' developing cognitive structures, and have traversed a route 
from discipline and paradigm to the university classroom. The underlying motivation in 
my program of research has been the hunt for explanatory principles of potential use 
within the field of higher education. 

Cognitive structure 

I began my research into learning in the university in 1977 with a research grant 
to examine the structure of knowledge in university courses as a means for 
understanding course development and the evaluation of student learning. The 
research process involved working with professors on courses they had developed to 
understand how the most important concepts in the courses were related. A concept 
was defined for this purpose as a unit of thought - an element of knowledge that 
enables us to organize experience (Donald, 1983). The concepts relevant to the 16 
courses studied ranged from 33 in the philosophy course to 170 in physical chemistry, 
with an average of 99 relevant concepts per course. From this set of concepts, the 
professors selected the main or linking concepts in the course, such as biological 
system and analysis of data in the microbiology course (Figure 1). 

Among the methods used to determine the relations between key concepts, the 
most useful method according to the professors was a tree structure or concept map 
which linked the most closely related concepts in order, so that link 1 was the closest 
link (Shavelson & Stanton, 1975). Figure 1 shows the hierarchical organization of the 
methods used in the microbiology course, with the most important concept, analysis of 
data, at the top, leading to a web of substantive concepts organized around biological 
system. In this laboratory course, the majority of concept relationships were causal or 
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procedural, that is, one concept caused or preceded the concept to which it was linked, 
suggesting a highly structured pattern of learning. Figure 2, from a n introductory law 
course on tort or wrongdoing, shows a series of factors such as risk that are 
subcategories of liability for fault, and that will in turn affect the recovery of damages. 
Each of these concepts {intentional tort to vicarious liability) has a logical or conditional 
relationship with recovery of damages. 

What did we learn from this research? First, the variety in kinds and numbers of 
concepts in a course suggested a wide difference in what students are expected to 
learn for the same credit. Second, in some disciplines, there are clear and well defined 
knowledge structures in which the concepts have essential ways of relating to each 
other. Other disciplines are less well structured (Frederiksen, 1984), and have looser 
organizations in which the major concepts are related more frequently on the basis of 
similarity than causation or logic. The implications for learning are that in some 
disciplines, the logic of the knowledge structure will control learning, while in others, 
students will have to seek a schema in a swamp. This would require very different 
approaches on the part of the learners (Donald, 1986; 1987). Given these results 
(Principles 1 and 2), the next step was to test learning and the development of students' 
cognitive structures in this same set of courses. 

1. The learning task varies substantially across courses; courses are not equivalent. 

2. Some courses have well defined and logically based knowledge structures which 
constitute the learning task. In other courses, knowledge structures based on 
similarity require a different form of learning. 



Students' cognitive structures 

During the next three years, we studied the comparative effects of student 
background variables and entering knowledge on student learning and achievement in 
the same 16 courses. At the beginning of the year I went to each class, and asked 

O 

ERIC 



7 



Indicators of success - AERA 2000 7 



students to define the set of key concepts for their course, and to provide information 
about their background preparation for the course. A research assistant attended 
classes and recorded the use of key concepts in class. At the end of the course, 
students were again asked to define the key concepts, and to do a tree structure of the 
key concepts. 

In the microbiology course, the professor had added four key concepts - 
recombination, deletion and complementation mapping, conjugation, restriction enzyme 
mapping, and restriction-modification systems. Although students knew least about the 
four recently introduced concepts at the beginning of the course, by the end of the 
semester they had made the greatest gains in knowledge of them. Student biology 
background was the strongest predictor of course achievement rather than general 
ability or specific knowledge of concepts. The study of this course showed that key 
concepts are frequently processes, that there is a need to update the knowledge 
structure within a brief period, and that background in the field is an important predictor 
of course achievement. 

In the law course, damage was the concept most frequently mentioned in class, 
but unintentional tortvjas the best known at the end of the course, with a very high 
understanding of the concept shown by the students. Students’ overall average the 
previous year had the highest correlation (.50) with final grade in the course, suggesting 
an ability effect, but their LSAT scores did not correlate significantly with grade. 

Students made the greatest gains in concept knowledge in this course compared with 
the other 15 courses. As most students in the course were in their first year of university 
and this was their first law course, their knowledge of the concepts was relatively low at 
the beginning of the course, and they showed a 33% gain in knowledge of key 
concepts. 

Overall there were several important findings from this set of studies (Principles 3 
and 4). One was that although students gained knowledge of the key concepts during 
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the course, on the whole they did not gain a great deal (on average 18%), and they left 
the course with at best a 67% knowledge of the most important concepts in the course. 
Knowledge of key concepts at the beginning of the course best predicted success in 
courses in the social sciences, but average the previous year tended to be the best 
predictor in the sciences, humanities and law. One explanation is that in the social 
sciences, students are learning a new vocabulary, so the course vocabulary will be a 
more important indicator than general ability. 



3. Overall, students do not acquire a precise knowledge of the most important concepts 
in their courses. 

4. In the sciences and humanities, previous achievement is a stronger predictor of 
achievement in a course than knowledge of key concepts in the course; in the social 
sciences, concept knowledge is a better predictor. 

These findings prompted two studies, one on different ways to portray knowledge 
structures, and the other on the acquisition of intellectual skills in postsecondary 
education. 

The portrayal of knowledge structures 

The main goal in this study was to determine the relative sensitivity of different 
portrayal techniques for representing knowledge structures within a course. We chose 
the law course from the previous studies, since the concepts showed a pattern of both 
hierarchical and logical relationships, important for potential generalization to other 
courses, and students had shown the greatest gain in concept knowledge. We selected 
a larger unit of analysis, the proposition, defined as "a statement which expresses 
relationships among concepts and which has a truth value", for comparison with the 
concept as a means of representing course material. We analyzed all course material to 
obtain 182 important propositions of which 73 were most important according to the 
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professor. We asked other experts (law professors who also taught tort) to rate the 
importance of the propositions for a course on tort. Students from the class were also 
interviewed to determine which propositions they considered important. 

Analysis of the relationship between the most frequently found concepts in the 
propositions with the 1 1 key concepts from the previous study showed that concepts in 
the propositions were subcategories of the key concepts, for example, apportionment of 
damages was a subcategory of recovery of damages (Donald & Nagy, 1985). The two 
other experts in the field agreed with the overall choice of the most important 
propositions of the professor teaching the course. Twenty-five propositions were agreed 
upon by all three professors as being important; these tended to be fundamental 
principles or definitions (14) which employed basic concepts and were more concrete. 
Disagreement arose when propositions were abstract, general or conditional. Students 
tended to choose the most familiar and specific propositions as important. What did we 
learn from this study? 



5. Within a discipline, agreement on the importance of basic knowledge structures, that 
is, propositions in an entry level course to the discipline, is greatest for fundamental 
principles or definitions. 

6. Students choose the familiar and specific as important in their courses. 



Intellectual skills in higher education 

The impetus for the study of intellectual skills was a discussion with the physics 
professor who participated in the study on course concepts. When I asked him why he 
thought student ability rather than concept knowledge was the best predictor of 
achievement in his course, he replied that it was because he was not teaching 
concepts, but rather how to analyze and synthesize. The study was designed to test 
models of skill acquisition, and to develop a comprehensive model of the intellectual 
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skills necessary for expertise in different disciplines. The first step was to examine the 
intellectual skills used at the postsecondary level as reported in research on critical 
thinking, problem solving, formal operations and creativity (Donald, 1985). Some 80 
operations or complex behaviors were identified, sorted and categorized by a set of 
experts in the field of postsecondary teaching and learning, to produce a working model 
of 30 thinking processes (Table 1) (Donald, 1992). Note that the thinking processes 
defining expertise include identifying the context, choosing relevant information, 
inference and verification. Critical thinking, in contrast, involves identifying assumptions, 
then inference and verification. All approaches to thinking processes involve verification. 

We interviewed professors across disciplines to ascertain to what extent students 
could be expected to have already acquired these skills, what skills were developed in 
the course, which evaluated, which skills the professor used and which were considered 
most important for students to acquire. Although there is a general tendency to think of 
biological science courses as being knowledge-laden and descriptive in nature, the 
biology course selected for study emphasized the development of thinking processes. In 
it, students are expected to identify the context, considered by the professor to be very 
important if extrapolations are to be successful (Table 2). Successful learning involves 
progress through alternating patterns of deductive and inductive thinking, with the use of 
inferential skills, particularly changing perspective. The professor pointed out that unless 
students understand the processes behind the derivation of biological knowledge, they 
cannot understand the limits of knowledge. He therefore gives them questions to be 
answered in their reports that require creative inference. In biology, inference and 
verification skills distinguish the expert from the novice. The final outcome is insight into 
how biological information is derived. Thus the focus is on knowledge, but from the 
perspective of its creation and limits, a powerful tool in any discipline. 
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Table 1, Working model of thinking processes in higher education 



DESCRIPTION (PS, SM) 

identify context (E) 

state conditions 

state facts 

state functions 

state assumptions (CT) 

state goal 

SELECTION (PS) 
choose relevant information (E) 
order information in importance 
identify critical elements 

identify critical relations 

REPRESENTATION (PS) 
recognize organizing principles 
organize elements and relations 
illustrate elements and relations 

modify elements and relations 

INFERENCE (E, H, CT, PS) 

discover new relations between elements 

discover new relations between relations 

discover equivalences 

categorize 

order 

change perspective 

hypothesize 

SYNTHESIS (PS) 

combine parts to form a whole 

elaborate 

generate missing links 

develop course of action 

VERIFICATION (E, H, CT, PS, SM) 
compare alternative outcomes 
compare outcome to standard 
judge validity 
use feedback 

confirm results 



Delineation or definition of a situation or form of a thing 
Establish surrounding environment to create a total picture 
State essential parts, prerequisites or requirements 
State known information, events that have occurred 
State normal or proper activity of a thing or specific duties 
State suppositions, postulates or propositions assumed 
State the ends, aims, objectives 
Choice in preference to another or others 
Select information that is pertinent to the issue in question 
Rank, arrange in importance or according to its significance 
Determine units, parts, components which are important 
Determine connections between things which are important 
Depiction or portrayal through enactive, iconic or symbolic means 
Identify laws, methods, rules which arrange in systematic whole 
Arrange parts, connections between things into systematic whole 
Make clear by examples, the parts, connections between things 
Change, alter or qualify the parts, connections between things 
Act or process of drawing conclusions from premises or evidence 
Detect or expose connections between parts, units, components 
Detect or expose connections between connections of things 
Detect or expose equality in value, force or significance 
Classify, arrange into parts 
Rank, sequence, arrange methodically 
Alter view, vista, interrelations, significance of facts or info 
Suppose or form a proposition as a basis for reasoning 
Composition of parts or elements into a complex whole 
Join, associate elements, components into a system or pattern 
Work out, complete with great detail, exactness or complexity 
Produce or create what is lacking in a sequence; fill in the gap 
Work out or expand the path, route or direction to be taken 
Confirmation of accuracy, coherence, consistency, correspondence 
Examine similarities or differences of results, consequences 
Examine similarities, differences of results to a criterion 
Critically examine the soundness, effectiveness by actual fact 
Employ results to regulate, adjust, adapt 
Establish or ratify conclusions, effects, outcomes or products 



E: Expertise, H: Hermeneutics, CT: Critical Thinking, PS: Problem Solving, SM: Scientific Method 
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Table 2. Professor's examples of thinking processes used in a biology course 
DESCRIPTION - experts use to describe new and unfamiliar situations 



Delineation or definition of 
a situation or form of a thing 


- data stated 


identify context 

Establish surrounding environment 
to create a total picture 


- very important if extrapolations are to be successful (inductive) 
-students have problems dealing with weights, measures and sampling 


SELECTION 

Choice in preference to another or others 

REPRESENTATION 

Depiction or portrayal through 
enactive, iconic or symbolic means 

INFERENCE 

Act or process of drawing conclusions 
from premises or evidence 

discover relations between 
relations 

Detect or expose connections 
between connections of things 


- all skills used in lab notebook 

- 2 to 3 page limit must be adhered to so students must select 

- decisions on how best to convey information 

- used in lab notebook 

- charts, graphs, diagrams 

- most important in inductive reasoning 

- distinguishes the expert from the novice thinker 

- course is first step toward advancement into expert level 

- ensured by questions to be answered in the final report 

- sample questions: What does each parameter tell you about this 
forest? How do the three parameters differ from one another in their 
biological impact? 



discover equivalences - in the sampling lab, students are asked to make discoveries about 

Detect or expose equality in value, a changing sample, and its relation to variability 

force or significance - sample question: Can you see other types of questions that might be 



categorize 

Classify, arrange into parts 


answered with data or density, frequency or dominance? 
- use sampling as a biological tool 


order 

Rank, sequence, arrange methodically 


- sample blood cells on a slide to do a differential count 


change perspective 
Alter view, vista. Interrelations, 
significance of facts or info 

hypothesize 

Suppose or form a proposition 
as a basis for reasoning i 

SYNTHESIS 

Composition of parts or elements 
into a complex whole 


- most important in inductive reasoning at the expert level within a 
broader framework. Biology has major impacts on modern life. Unless 
students understand the processes behind the derivation of biological 
information, they cannot understand the limits of knowledge. 

- students are asked questions that demand that they hypothesize, 
predict, and discover, for example. What, if anything, can you predict 
about a canopy’s forest composition in 200 years? 

- lab notebook is evaluated for new hypotheses 

- experts apply biological Information 


VERIFICATION 

Confirmation of accuracy, coherence, 
consistency, correspondence 


- ability to be critical of and to question results and conclusions 

- expert uses simultaneously with other processes; influences how 
other processes used (deductive thinking) 
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In the law course, the processes most focused were called the methodology of 
legal judgment: recognizing organizing principles (representation), discovering new 
relations between elements and categorizing (inference), developing a course of action 
(synthesis), and verification. The professor described the main goal of the course as 
developing students' ability to use the legal system which requires developing a high 
degree of analytical ability. The study produced the following general results. 



7. A working model of thinking processes can be used across disciplines to determine 
the kinds of thinking processes developed in courses. 

8. Inference and synthesis were most frequently used by professors to describe the 
kinds of processes developed in their courses, although different terms were used 
as blanket descriptions of the process. 



Professors' expectations of students' ability to think 

We next expanded our context to examine the expectations of learning held by 
professors and students in selected disciplines, including the knowledge, thinking 
processes, and attitudes expected to be acquired by students (Donald, 1988, 1990). We 
investigated students' perceptions as well as their professors'. Thirty-six professors 
were selected as model teachers and researchers from the University of Western 
Ontario, Cambridge University and Stanford University. They represented six matched 
pure and applied disciplines in the physical and social sciences and the humanities; 
physics, engineering, psychology, education, English literature and English language. 
This yielded a sample of six experts in each disciplinary area, interviewed individually in 
structured but open interviews. 

One part of the study consisted of asking what their expectation of students' ability 
to think in their courses was; two-thirds replied that they expected their students to think 
logically and to reason with abstract propositions, more in the natural and social sciences 
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than in the humanities (Donald, 1988). Most defined logic in terms of an interrelation or 
sequence of facts or events seen as predictable, a definition of logical empiricism. An 
English professor explained that the analysis of text does not require a step by step 
logical pattern but rather the creation of a series of tentative hypotheses. Over half the 
professors expected their students to think independently, and just under half expected 
abstract thinking from their students. Overall, the social science professors had the 
greatest tendency to expect thinking abilities in their students, with natural science 
professors equally expectant in the areas of logical thinking and reasoning with abstract 
propositions, but less expectant about abstract thinking. The humanities professors had 
lower expectations of these abilities: they appeared to be looking for something other 
than logical thought from their students. The disciplinary groups most expectant of 
thinking abilities were the engineers, who focused on the development of problem 
solving skills in their program, but who were least interested in abstract thinking, and the 
educators and psychologists, who had a more general interest in all of the thinking 
abilities. 



9. Logical thinking was expected of students in their courses by two-thirds of the 
professors (all in engineering and education). 

10. Professors' expectations of their students' ability to think were greater in education, 
psychology and engineering than in other disciplines. 



The student learning task 

In the comparative study of student learning across disciplines, in 1990 a sample 
of professors from Monash University in Australia was added to those in Canada, the 
United Kingdom, and the United States. A class from each discipline in each university 
was observed, and students from the course were interviewed for their perceptions of 
what and how they learn in their courses. Their perceptions were then compared to their 



Indicators of success - AERA 2000 1 5 



professors'. Learning in three of the six disciplines - physics, engineering and 
psychology - is compared here. 

The physics professors considered the learning task in physics to be difficult, 
primarily due to the sheer weight of the theory, but also because physics is often 
counterintuitive (Donald, 1993; 1994). The professors had a highly intellectual, content- 
oriented view of instruction: in class, emphasis was on covering course material. Direct 
teaching of problem solving skills did not tend to occur in class. Instead, it was the role 
of assignments, tutorials, or laboratories to develop these skills; in response to this 
situation, students commented that their learning took place in laboratories or 
supervisions rather than in the classroom. In contrast to their professors’ view of 
learning as an intellectual exercise, students in introductory physics courses tended to 
think of learning in the course as a matter of acquiring knowledge. More advanced 
students also made the distinction between learning concepts in lectures and 
developing thinking processes through problem solving. 

Professors' and students' views of the learning process in engineering were 
highly consistent, focusing on the problem solving process (Donald, 1991b; 1994). 
Professors were very student centered in their approach to instruction; they emphasized 
students' acquisition of learning strategies. In more advanced courses, design and 
communication skills were added to the repertoire. In response to their program, in their 
first year, engineering students talked about survival skills, including ‘not doing all night 
sessions', 'checking solutions', or gaining perspective. In advanced courses, the 
instructional methods also had changed, with greater emphasis on discussion and 
greater opportunity to ask questions. Feeling less rather than more prepared for their 
courses than first and second year students, third and fourth year students were attuned 
to possible 'managerial' solutions, in which one reflects, then tries out a solution on a 
colleague, and if that has a good result, talks until 'something feasible comes out.' 
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Psychology professors took a third approach, that different courses supplied 
students with different intellectual skills (Donald, 1994). Instructional methods were 
varied, even within a course. Introductory courses were primarily vocabulary oriented, 
as students were entering a new field. Professors spoke more about graduate training 
than physics or engineering professors had, where apprenticeship and metacognitive 
skills became more important. Psychology students quickly recognized that they were 
learning to think, and gave examples such as writing up experiments, creating models, 
or learning how to think critically: these were learned through a combination of lectures, 
labs, specified reference material and discussion. 



11. Disciplinary differences in emphasis on what is learned are reflected in instructional 
method and in students' attitudes toward the program. 

12. If professors emphasize knowledge, students will approach the learning task as if it 
is one of acquiring knowledge; if thinking, students will interpret the task as learning 
to think. 



Students' learning experience 

The study of student learning across disciplines showed that professors and 
students could define the kinds of thinking developed in courses, but no study had been 
done of the way in which thinking processes were modeled or developed in class. In the 
next study, introductory level courses in the same disciplines were selected to be 
audited by a participant observer - a graduate research assistant who attended classes, 
labs, and conferences, and kept a record of class experiences. Lectures were 
audiotaped and a classroom observation matrix was used to examine the pedagogical 
strategies used, and the kind of thinking processes and learning strategies developed in 
each class. 
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The reports of the participant observers gave a perspective of learning in 
considerable contrast to those provided in interviews in the previous studies. From 
reading the log, for example, in the introductory psychology class, it became clear that 
students without a background in science tended to ignore scientific concepts instead of 
spending more time on them. The students' social life was paramount, and many 
students appeared to discover the need to think only after receiving midterm results. 
Overall, study of the classroom experience revealed that students, although prepared in 
terms of subject matter background, lacked the strategies to take maximum advantage 
of the university learning situation. This phenomenon suggests an explanation for the 
relatively low (18%) increase in concept knowledge across courses. 



13. The perspective of learning offered by participant observation in courses was 
broader than that offered by interview and visitation, and identified the important role 
of student life in the classroom. 

14. Learning strategies were underdeveloped in many students in introductory classes, 
although students appeared to be prepared in terms of subject matter background. 



Students' orientation to learning 

Our studies of the student learning experience alerted us to the need to better 
understand the student population in terms of more general motivational and behavioral 
indices. We therefore examined models of student orientation and motivation and their 
ability to predict success in courses (Ramsden, 1992). We began by asking students to 
describe their conceptualization of learning, learning strategies and educational 
expectations, in a questionnaire followed by interviews. Although students rated 
themselves as being well prepared for their courses, they did not rate their study skills 
as highly (Donald, 1995). Major differences were noticeable in how students in different 
programs spent their time, for example physics students spent more time studying and 
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doing homework and less time socializing and partying than students in the English 
literature course; English students reported spending as much time socializing as they 
did studying. Students exhibited a deep or meaning orientation to learning in their 
reasons for attending university, but they also attached importance to improving their 
chances of finding a job. 



15. Students have a meaning orientation to learning and an achieving orientation in 
their reasons for attending university. 

16. Students differ more in the way they spend their time than in their orientation to 
learning. 



The effect of the learning climate on higher order learning 

Our research had suggested that different learning contexts interacted with 
student characteristics to produce different learning effects, and specifically that some 
learning contexts challenged students to think more than others. We began offering 
academic seminars to first year university students in 1996, and as part of the 
evaluation of the project, asked them by means of questionnaires and interviews what 
emphasis they perceived was placed on higher order learning in the university and how 
their experience in the seminar compared to that in other courses. Large differences 
occurred between what students saw occurring in their seminar and in their other 
courses. In the seminars, students were challenged to think for themselves, but were 
neutral about their other courses providing challenge (Donald, McMillan-Davey & 
Denison, 1999). 

They perceived that assessment of learning in the seminars tested their 
understanding rather than their ability to memorize, but in large classes, they were 
again neutral about whether their learning was adequately assessed. The differences 
between the climate in the seminars and their other courses extended to how the 
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students interacted with each other and with their professor. In the seminars, students 
more often helped each other in understanding difficult materials, got to know each 
other well, and were enthusiastic about participating in class activities. They reported 
that the professor spent more time talking informally with them before or after class, and 
was more willing to assist them outside of class. The difference in climate extended 
more generally to the enjoyment students felt in going to class and, more importantly, to 
the effort they put into what they did in class. 



17. Students are challenged to think in seminars more than they are in large classes; 
assessment of their learning coincides with this emphasis. 

18. Student cohesiveness and satisfaction are greater in seminars than in large 
classes: access to the instructor is greater. 



Synthesis of research results 

The results of these studies fall into three nested categories. At the most general 
level are findings about methods and lines of convergence across institutions and 
disciplines. These results describe variables in our context as faculty developers. At a 
more specific level, we can examine student learning and compare student experience. 
Nested within are the effects of differences in learning contexts, for example, in large 
and small classes. They serve as cautions and conditions that we need to take into 
consideration. These principles give us further insight into potential goals and conditions 
that we as faculty developers need to take into account, and the indicators of success 
we could use to ensure that learning is taking place. 

Our context as faculty developers 

Perhaps the most expected finding is that experts in a discipline agree most 
readily on the importance of fundamental principles or definitions in a course (Principle 
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5). Of some concern is the level of disagreement over abstract, general or conditional 
propositions in a course. This raises a fundamental curriculum issue: How variable is 
course content? How accountable are we for what we are teaching? In the past, 
academic freedom has protected course material; with the advent of electronic courses, 
how much attention needs to be paid to what constitutes a course? 

A working model of thinking processes could be applied across disciplines 
(Principle 7). This supports the hypothesis that we have something in common in our 
attempt to develop students' thinking processes in the university. The preference for the 
development of inference and synthesis (Principle 8) further specifies an important 
outcome of postsecondary education. The thinking processes in the model could be 
used as criteria for course goals. Another important methodological finding was that 
participant observation in courses provided a very different perspective of student 
experience and learning than interview and visitation (Principle 13), reminding us that 
students have lives to live in addition to scholarly inclinations. How important other 
aspects of students' lives are, and the relative weight given to scholarly versus social 
activities and whether we can change this or use this knowledge, for example, in peer 
learning situations, merits further study. 

Students 

The findings on students are of two kinds, ones that describe them as they enter 
university, and others that describe interactions between them and the university milieu. 
Students come to us with a meaning orientation to learning and an achieving 
orientation: they want to learn and they want to succeed (Principle 15). Even so, there is 
a wide range in the way they spend their time (Principle 16). They appear to be 
prepared in terms of subject matter background but not in their learning strategies 
(Principle 14). We expect them to be able to think logically (Principle 9), but they choose 
the familiar and specific as important in their courses (Principle 6) and do not acquire a 
precise knowledge of the most important concepts in their courses (Principle 3). 



Indicators of success - AERA 2000 21 



They interpret the learning task as we describe it: if we emphasize knowledge, 
students will approach the learning task as if it is one of acquiring knowledge; if thinking, 
students will interpret the task as learning to think (Principle 12). These findings suggest 
that we need to provide students with clear expectations about what is important to 
learn and, more specifically, that they are expected to think. We also need to provide 
them with strategies for learning, including the amount of time it takes to learn, and with 
a learning climate where they are helped to learn. This will require emphasizing the 
structure of learning, orienting students to the new challenges that postsecondary 
education provides, and changing our instructional context. 

Differences across learning contexts 

The first major difference to be discovered across courses was the variation in 
the learning task, particularly in the number and kinds of concepts students are 
expected to learn, but also in how concepts are related in the course (Principles 1 & 2). 

If courses are not equivalent, then different standards must be assumed or assurance of 
equivalent value becomes an institutional priority. Differences in instructional methods 
reflect differences in the learning task (Principle 11). What we do not know is whether 
these are necessary or vicarious differences. 

In the sciences and humanities, students' general ability is a stronger predictor of 
achievement in a course than more specific knowledge of concepts in the discipline; in 
the social sciences, concept knowledge is a better predictor (Principle 4). Does 
explanation of this difference lie in a new vocabulary, in the familiarity of terminology of 
more established disciplines in the general culture, or in the effect of prior educational 
exposure to these areas of study? Do the social sciences offer a truly different kind of 
education? Professors' expectations of entering students' ability to think were greater in 
social science fields such as psychology and education and in engineering than in other 
disciplines (Principle 10). Research shows that students make differential increases in 
inferential skills across the disciplines, favoring psychology (Lehman, Lempert, & 
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Nisbett, 1988; Nisbett, Fong, Lehman & Cheng, 1987; Ratcliff, 1988). This adds weight 
to the hypothesis that a different kind of education is occurring. 

The kind of teaching method also affects learning. If students think in seminars 
more than they do in large classes, postsecondary institutions need to take account of 
the fact that smaller classes provide a better education (Principle 17). Assessment of 
student learning in the seminars coincides with this emphasis and leads to a more 
productive learning experience overall. Student cohesiveness and satisfaction are 
greater in seminars than in large classes, and access to the instructor is also greater 
(Principle 18). Seminars lead to communities of learning; large classes lead to anomie. 
What is our responsibility as faculty developers to our institutions with reference to the 
differential effects of learning contexts? 

Indicators of success 

These principles suggest indicators of success that represent major challenges 
for us as faculty developers. The similarities across contexts provide the first three 
indicators. The first is the appropriate content in a course. What is the conceptual 
structure of the course? Are fundamental principles and definitions being taught? Would 
other experts in the field agree with these principles? Are students learning the 
concepts and propositions? We begin our course design and teaching workshops with a 
day in which our professors create a concept map that will explain the major concepts in 
a course and their relationship. Throughout the week, they may adapt or alter this 
concept map so that it provides a conceptual road map for their students that is 
consistent with their instructional and evaluation methods. 

In keeping with the general goal of postsecondary education to increase 
students’ ability to think, the second indicator of success is whether students’ have 
learned to think in a new way from their experience in the course. Accomplishing this 
goal will require explaining to students the difference between knowledge crunching and 
higher order learning, whether defined in terms of critical thinking, problem solving, or 
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expertise. We spend the second day of our course design and teaching workshop using 
models of thinking processes to establish higher order learning outcomes. 

The third indicator is the extent to which students incorporate their learning in 
courses into their studies more generally and into their lives as scholars. Providing 
opportunities for students to extend their learning by means of extracurricular activities 
or field experiences is a program responsibility and requires extended planning by a 
curriculum committee. It also requires interaction at the campus level to ensure that 
these kinds of activities are supported by the campus as a whole. As faculty developers, 
it then becomes our responsibility to be proactive in establishing that kind of campus 
environment. 

Responding to students’ needs sets an additional series of indicators. We need 
to be clear in our expectations of what a successful student is, and enable our students 
to reach these goals. We need to provide students with strategies for learning, by 
means of introductory courses that include learning and study strategies appropriate to 
the discipline, by supplementary learning skills courses, by establishing peer study 
groups, and by means of advisors who can solve problems before they become crises. 

The third set of indicators are more arcane and might appropriately be used to 
guide our own research. We need a better understanding of the variety of instructional 
goals and methods across disciplines and the extent to which they can be transferred. 
For example, would laboratories in social science courses lead to greater higher order 
learning? What methods and approaches best support higher order learning? If 
seminars challenge students to think in a manner that lectures do not, and eliminating 
lectures is unfeasible, how do we make lectures more like seminars? 

These indicators of success broaden our responsibilities and require a proactive, 
one might say, diplomatic stance. But no longer the bearers of at times distasteful news 
to faculty, we can help them to be proactive in solving instructional problems and in 
aiding their students to have a meaningful learning experience. 
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